Designing a trust evaluation model for open-knowledge communities

The openness of open-knowledge communities (OKCs) leads to concerns about the knowledge quality and reliability of such communities. This confidence crisis has become a major factor limiting the healthy development of OKCs. Earlier studies on trust evaluation for Wikipedia considered disadvantages such as inadequate influencing factors and separated the treatment of trustworthiness for users and resources. A new trust evaluation model for OKCs—the two-way interactive feedback model—is developed in this study. The model has two core components: resource trustworthiness (RT) and user trustworthiness (UT). The model is based on more interaction data, considers the interrelation between RT and UT, and better represents the features of interpersonal trust in reality. Experimental simulation and trial operation for the Learning Cell System, a novel open-knowledge community developed for ubiquitous learning, show that the model accurately evaluates RT and UT in this example OKC environment. Introduction Recently, open knowledge communities (OKCs) have become more popular. OKCs can be used as knowledge management tools and virtual learning environments for learners (Zeng, 2011). According to the content whether it allows collaborative editing, OKCs are divided into two categories. One is represented by the Wikipedia (http://www.wikipedia.org/) and the Learning Cell System (LCS, http://lcell.bnu.edu.cn) where any valid user can create new knowledge and coedit existing knowledges. The other one is represented by Baidu Zhidao (http://zhidao.baidu. com/), where users have no rights to edit knowledges created by others. They can usually view the existing knowledges and post comments. In this paper, OKCs are specifically refered to the former. OKCs have inherent advantages in attracting user participation, encouraging collaboration and promoting the sharing of knowledge. However, openness also has side effects. A major problem currently underlying OKCs is information reliability (Yang, 2012). As the most popular online encyclopedia and an excellent example of OKCs, Wikipedia encountered the crisis of confidence inevitably (Lever, 2005; Luo & Fu, 2008; Seigenthaler, 2005). Wang (2009) pointed out that the feature of completely open content editing and organization of Wikipedia resulted in considerable British Journal of Educational Technology Vol 45 No 5 2014 880–901 doi:10.1111/bjet.12083 © 2013 British Educational Research Association doubt as to its quality and reliability. In recent years, Wikipedia has evolved a set of collaborative mechanism during the course of its development (Aniket & Robert, 2008), including dialogue pages, historic pages, recognition of quality problems in the community and quality control of entries. Although Wikipedia has introduced new mechanisms to improve its editing process, the quality of its data entries remains seriously questioned (Dondio, Barrett, Weber & Seigneur, 2006; Wang, 2009). Due to the vast, complex and uncontrol features of the Internet, it is rather difficult to clean up those inferior resources and malicious users in OKCs. The central problems are how to evaluate the credibility of users and the knowledge they create and how to help users identify reliable information in OKCs. Trust modeling has greater value for fixing the above problems (Lucassen & Schraagen, 2011). Some researchers have begun to study the trust evaluation models in Wikipedia (Javanmardi, Lopes & Baldi, 2010; Maniu, Abdessalem & Cautis, 2011) and in virtual learning communities (Wang & Liu, 2007). Through trust evaluation models, users can find the right and reliable learning resources and connect with the right and reliable people. It is extremely useful for improving learning experience and promoting effective learning in OKCs. Practitioner Notes What is already known about this topic • The confidence crisis for open-knowledge communities (OKCs) is emerging. • Scholars begin to apply the idea of trust in social networks to solve confidence crisis in OKCs. • The design of trust evaluation models and the application to OKCs are still at an early stage. • Current trust evaluation models are constructed simply based on user interaction data (eg, the editing history of a paper) and to apply such trust to the evaluation of resource quality. What this paper adds • We developed a new trust evaluation model named two-way interactive feedback model for evaluating trust in OKCs. • The model takes into account editing history and typical interactions in OKCs, thereby improving the modeling accuracy. • The model considers the dynamic interrelation between user trustworthiness (UT) and resource trustworthiness (RT), and models this interrelation by iterative crosscomputation. • The model adopts useful factors in existing trust evaluation models for network communication and electronic business (eg, the time-decay effect and punishment factor), thus better representing the interpersonal trust relation in reality. Implications for practice and/or policy • Introducing trust mechanisms is an effective means to solve crises of confidence in OKCs. OKCs should add trust evaluation functions to help users assess the trustworthiness of knowledge and other users. • To ensure model comprehensiveness and integrity, a successful trust evaluation model should consider various typical interaction operations under Web 2.0. • Instead of treating RT and UT separately, an effective trust evaluation model should consider their interrelation and model them by cross-computation. A trust evaluation model for open-knowledge communities 881 © 2013 British Educational Research Association The main objective of the present study is to redesign a new trust evaluation model for OKCs. Research questions can be stated as follows: (1) What influencing factors should be taken into account while designing the trust evaluation model for OKCs? (2) How to set the computing methods for computing users’ and resources’ trustworthiness? (3) How to prove the validity of the trust evaluation model for OKCs? Literature Features of OKCs Por (1997) described knowledge communities as self-assembled knowledge-sharing networks joined by knowledge islands. OKCs are network communities assembled by various interactions (human–human, knowledge–knowledge and human–knowledge) for the purpose of creating, spreading and sharing knowledge. Common network communities are characterized by communicating and sharing information. These communities satisfy social and daily needs of users and provide a sense of belonging (Wang & Guo, 2003). Besides the characteristics mentioned above, OKCs have some other features as follows: User diversity and variation in knowledge quality The openness of OKCs leads to user diversity and variation in knowledge quality (Wang, 2009). Although most users are benign, there are malignant users who spread inferior knowledge and post malicious comments to earn community scores and ranks. Similarly, although group collaboration produces high-quality knowledge, inferior and untrustworthy knowledge also arises side by side. Multiple interaction modes As an ecosystem, an OKC has two key species including the user and the knowledge (Yang & Yu, 2011). Interaction is an essential means for information flow in the ecosystem, and this includes interactions between knowledge (eg, a citation, link), between users and knowledge (eg, browsing, comments, subscription, editing and bookmarking) and between users (eg, collaboration, reply, invitation and sharing). User–knowledge interactions In an OKC, users and knowledge are interrelated and interactive. Users produce, consume and transmit knowledge. Knowledge is the essential “food” consumed by users in an effort to selfupgrade (ie, gain skills and knowledge) (Yang & Yu, 2011). The trustworthiness of a user directly affects the trustworthiness of his or her contributions (created/shared knowledge). Conversely, the trustworthiness of knowledge shared by a user also affects the trustworthiness of the user himor herself. Partial resemblance to real communities As a special form of virtual communities, OKCs resemble real social communities in certain respects. For example, interpersonal trust is affected by time and social events. Trust evaluation models Social trust is a belief in the honesty, integrity and reliability of others (Marsh, 1994). Trust evaluation model establishes a management framework of trust relationship between entities, involving expression and measurement of trust, and comprehensive calculation of trust value (Zhou, Pan, Zhang & Guo, 2008). With the emergence of a confidence crisis for OKCs, researchers began investigating solutions from the perspective of trust. Most recent studies on trust in OKCs have focused on Wikipedia because of its popularity. Adler et al (2008) developed a method to assign trust values to Wikipedia articles according to the revision history of an article and the reputation of the contributing authors. Javanmardi et al 882 British Journal of Educational Technology Vol 45 No 5 2014 © 2013 British Educational Research Association (2010) designed three computational models of user reputation based on user edit patterns and statistics extracted from the entire English Wikipedia history pages. Moturu and Liu (2009) proposed a model to calculate the trustworthiness of a Wikipedia article according to the degree of dispersion of the feature values from their mean. Lucassen and Schraagen (2010) showed that textural features, references and images are key indicators of the trustworthiness of Wikipedia articles. Maniu et al (2011) constructed a web of trust from the interactions of Wikipedia users, and analyzed user trustworthiness (UT) and its effect on readers and article classification. Halim, Wu and Yap (2009) proposed a method to improve the trustworthiness of Wikipedia articles according to credential information provided by a third party (eg, OpenID and OAuth). In their method, a third-party signature is added to all articles edited by a user, and the trustworthiness of an article is calculated by taking into account user information, such as education level, professional expertise or affiliation. Korsgaard (2007) developed a proxy Recommender System for Wikipedia, which allows users to rate articles and thus guide other users in terms of the trustworthiness of articles and users. In summary, the above studies present two general approaches for carrying out trust research for OKCs (as represented by Wikipedia). The first approach attempts to construct a trust model simply based on user interaction data (eg, the editing history of an article) and to apply such trust to the evaluation of article quality. The second approach calculates UT and constructs a user trust network based on user interactions or user information from other sources. Both approaches ignore the intrinsic link between UT and information trustworthiness. However, UT can be an important factor of content trustworthiness and, in turn, content trustworthiness influences UT. For example, an article on instructional design written by an educationalist is usually credible, and publishing multiple excellent articles on instructional design enhances the trustworthiness of this author. In addition, the input for trust calculation should not be limited to editing history and contribution-based user interactions. It needs to include these common and abundant interactive data under the Web 2.0 framework, such as comment, subscription, bookmark, invitation and so on. Since Marsh (1994) first introduced trust in social networks to computer networks, trust evaluation models have been widely studied and used in network communication (Denko, Sun & Woungang, 2008; Tian, Zou, Wang & Cheng, 2008; Yu, Zhang & Zhong, 2009) and electronic business (Jones & Leonard, 2008; Li & Wang, 2011; Wang, Xie & Zhang, 2010). Compared with current models for OKCs, these models consider the effects of time decay on trust evaluation, and some have introduced punishment factors to differentiate the effects of positive versus negative interactions (Liu, Yau, Peng & Yin, 2008; Wang et al, 2010). These design details suit the features of social trust in reality and provide a valuable reference for developing appropriate trust evaluation models for OKCs. Overall, the design of trust evaluation models for OKCs are still at an infancy stage. There are three major drawbacks: (1) ignoring the intrinsic link between UT and information trustworthiness, (2) excluding some key interactions (eg, comment, subscription, bookmark and invitation) that affect trust calculation and (3) unrepresenting the interpersonal trust relation in reality (eg, the time-decay effect and punishment factor). Trust evaluation framework For an OKC, the objects to be evaluated include its users and knowledge. Accordingly, their credibilities should be evaluated separately. Knowledge may appear in different forms, such as entries in Wikipedia, items in Hudong and Knol pages in GoogleKnol. Despite their various forms, digital resources act as the carrier of knowledge in all OKCs. Therefore, in the present work, knowledge and a “learning resource” are considered synonyms; similarly, knowledge trustworthiness evaluation purports to evaluate the trustworthiness of learning resources. A trust evaluation model for open-knowledge communities 883 © 2013 British Educational Research Association Dong (2010) proposed qualitative method and quantitative method to design trust evaluation models in Grid Service. According to the quantitative method, four steps should be finished in order: (1) analyze factors that influence trust computing, (2) set basic assumptions for constructing model, (3) build trust evaluation model and (4) develop trust computing methods guided by the model. Because of its clear flow and operability, we will use this quantitative method to design the trust evaluation model for OKCs. Analyze influencing factors There are lots of factors we should consider carefully for constructing the trust evaluation model for OKCs. However, which factors actually affect the trustworthiness is the first question to figure out. By extending factors properly in current trust evaluation models for Wikipedia (Adler et al, 2008; Korsgaard, 2007; Maniu et al, 2011), the influencing factors of resource trustworthiness (RT) and UT are identified. Now we will discuss factors affecting RT and UT. Factors affecting RT RT can be evaluated in two ways. First, the systems provide trust evaluation functions allowing users to score RT directly, which is called direct evaluation. Second, RT can also be evaluated through other rich user–resource interaction data indirectly, which is called indirect evaluation. Direct evaluation. There is no universal system for the visible evaluation of RT. OKCs currently use different trustworthiness indicators according to their individual features and requirements. Wikipedia now assesses articles in terms of reliability, objectivity, completeness and writing formality. The LCS assesses content in terms of accuracy, objectivity, completeness, citation formality and updating timeliness. Baidu Baike and Hudong Baike grade articles with up to five stars and invite readers to vote to what extent “This is helpful.” Indirect evaluation. Invisible evaluation relies on records of user–resource interactions, such as collaborative editing, subscribing, bookmarking, browsing and citing. Obviously, different OKCs support different interaction modes. To some extent, user–resource interaction reflects user perception of RT. For example, an increase in user subscription of resource A suggests its attractiveness and trustworthiness. Factors affecting UT The trustworthiness of an OKC user is determined by the average trustworthiness of the resources that he or she created and also by his or her interactions with other users. Different interaction modes reflect invisible evaluations between users. Common factors of UT include the following. Trustworthiness of user contributions. The trustworthiness of resources created by a user affects his or her own trustworthiness. For instance, if user A has created high-quality and reliable resources, his or her trustworthiness is increased. Number of invitations/cancellations for collaboration. An invitation from user A to user B can be considered a positive vote for B by A. Conversely, cancelling an invitation is deemed a negative vote. If many users have invited user B to collaborate on resource editing, then user B has high trustworthiness. Number of additions/cancellations as a friend. User A adding User B as a friend is considered a positive vote for B by A. Conversely, cancelling friendship is regarded as a negative vote. If many users have added user B as a friend, user B has high trustworthiness. Number of accepted/rejected content revisions. The acceptance of content editing made by user A is a positive vote for A, and the rejection of editing is a negative vote. A greater probability of editing acceptance relates to higher trustworthiness. 884 British Journal of Educational Technology Vol 45 No 5 2014 © 2013 British Educational Research Association Set assumptions Inspired by the research findings of trust evaluation models in network communication and electronic business (Denko et al, 2008; Jones & Leonard, 2008; Wang et al, 2010), the following assumptions are determined to guide the model construction for OKCs. One of the most important principles is to reflect the trust relationship in the real society as much as possible. Time-decay effect It is assumed that trust decays with time (Li & Wang, 2011). The effects of user–resource and user–user interactions on trust are both time-dependent and time-limited. The effects of an interaction operation on trust decay with time. Thus, compared with earlier interactions, recent interactions have greater effects on trust. Differential effect It is assumed that interactions between one object (resource/user) and different users are associated with different effects on the trustworthiness of that object. Interactions with highly trusted users contribute better to the trustworthiness of that object and vice versa. Participant size effect Evaluations made by a large number of participants are assumed to be reliable. If a large number of users voted for a resource (in visible evaluation), the voting result is considered an accurate indication of the trustworthiness of that resource. Conversely, if few users voted for that resource, the result is regarded as uncertain or unreliable. Two-way interactive effects If a resource is cited, recommended, subscribed or bookmarked many times, it is considered well received and trusted by users. Similarly, if a user has made many accepted revisions, has created many credible resources, or has been invited to collaborate and been added as a friend many times, then he or she is considered well accepted by other users and his or her operations are thus believed to be more credible. Build trust evaluation model Employing the above influencing factors, analyses and assumptions, an OKC-oriented trust evaluation model (see Figure 1)—a two-way interactive feedback model (TIFM) is constructed. The TIFM includes two core interactive components: UT and RT. In Figure 1, information boxes on both sides explain the factors affecting the two components, and the oval in the center lists the four assumptions behind the evaluation. As opposed to trust between peer nodes in P2P networks, trust is defined as global trust in this TIFM. The trustworthiness of a source represents the overall trustworthiness evaluation made by all community users for this resource. Likewise, Figure 1: Framework of the two-way interactive feedback model as a trust evaluation model A trust evaluation model for open-knowledge communities 885 © 2013 British Educational Research Association the trustworthiness of a user means the overall trustworthiness evaluation made by all other users for that particular user. Develop computation methods In this part, the second research question is answered by developing the computation methods of RT and UT under the guidance of TIFM. Trustworthiness: definition and calculation Trustworthiness can be expressed by discrete values or continuous numbers. Discrete representation of trustworthiness resembles the features of human perception but lacks computability. Continuous representation, on the other hand, is amenable to modeling and computation, but it is not straightforward for users to rate UT/RT. In the TIFM, trustworthiness is expressed as continuous real numbers in the range (0, 1) and mapped into five trustworthiness ranks (full, strong, medium, weak, very weak) to provide users with a straightforward assessment of UT/RT. RT RT includes direct RT (DRT) and indirect RT (IRT). DRT is calculated from direct (ie, visible) user evaluations, and IRT is calculated from records of human–resource interactions. Shortly after creation of a resource, only a small number of users are expected to have posted direct evaluations of this resource. Therefore, at that stage, the contribution (or weight) of DRT to RT should be low. The weight (w) is a function that increases with the number of direct trust evaluations as the independent variable. It serves to dynamically adjust the relative importance of direct trust evaluations in RT calculation. With an increasing number of DRT evaluations, w increases and DRT accounts for an increasing proportion of RT. Equation 1 is used to calculate the resource trustworthiness by combining DRT and IRT. RT w DRT w IRT = × + − ( )× 1 . (1) Furthermore, DRT can be expressed by an indicating factor set denoted by DRT indicating factors (DIFs) = (dif1, dif2, dif3 . . . difn), where n is the total number of indicators and difi represents the i indicator. Individual OKCs focus on different aspects, and their DIFs are accordingly individualized. Even for a specific OKC, the DIF can be dynamically adjusted when required. Here, we give a relatively general DIF for RT: DIF = (content accuracy, content objectivity, content completeness, citation formality, updating timeliness). Correspondingly, the weight set for this DIF can be given as DW = (DW1, DW2, DW3 . . . DWn), where DWi represents the weight of the i indicator among all indicators (ie, ΣDWi = 1). The purpose for creating Equation 2 is to provide a formula to calculate DRT, which is part of Equation 1.